improve robustness of the simple runscript #1307

sakogan · 2021-04-13T18:18:25Z

Running the simple runscript with a task that does not have datasets for all the three phases (train/test/val) results in error. E.g., the following command:

python jiant/proj/simple/runscript.py run --run_name simple --exp_dir ${EXP_DIR}/ --data_dir ${EXP_DIR}/tasks --hf_pretrained_model_name_or_path roberta-base --tasks squad_v1 --train_batch_size 16 --num_train_epochs 3

results in the following error:

File "jiant/proj/simple/runscript.py", line 267, in <module>
    main()
  File "jiant/proj/simple/runscript.py", line 259, in main
    run_simple(args, with_continue=False)
  File "jiant/proj/simple/runscript.py", line 151, in run_simple
    tokenize_and_cache.main(
  File "jiant/proj/main/tokenize_and_cache.py", line 203, in main
    examples=task.get_test_examples(),
  File "jiant/tasks/lib/templates/squad_style/core.py", line 397, in get_test_examples
    return self.read_squad_examples(path=self.test_path, set_type=PHASE.TEST)
  File "jiant/tasks/core.py", line 150, in test_path
    return self.path_dict["test"]
KeyError: 'test'

This fix reads the config file of the given task and skips phases not specified in that config.

Skip task phases (e.g., "test") not specified in the config of a task (e.g., "squad_v1")

codecov · 2021-04-13T18:23:06Z

Codecov Report

Merging #1307 (bd488a8) into master (65888b4) will decrease coverage by 0.00%.
The diff coverage is 20.00%.

@@            Coverage Diff             @@
##           master    #1307      +/-   ##
==========================================
- Coverage   48.47%   48.46%   -0.01%     
==========================================
  Files         163      163              
  Lines       11216    11220       +4     
==========================================
+ Hits         5437     5438       +1     
- Misses       5779     5782       +3

Impacted Files	Coverage Δ
jiant/proj/simple/runscript.py	`44.53% <20.00%> (-0.64%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 65888b4...bd488a8. Read the comment docs.

jiant/proj/simple/runscript.py

jeswan

Thanks @sakogan! Just ran into this issue so this was great timing.

sakogan · 2021-04-14T17:59:32Z

No problem! You may also want to update the Bash example in the Quick Introduction of the Readme to use "hf_pretrained_model_name_or_path" instead of "model_type".

improve robustness of the simple runscript

fe64a68

Skip task phases (e.g., "test") not specified in the config of a task (e.g., "squad_v1")

sakogan requested review from jeswan and zphang as code owners April 13, 2021 18:18

jeswan reviewed Apr 14, 2021

View reviewed changes

jiant/proj/simple/runscript.py Outdated Show resolved Hide resolved

jeswan reviewed Apr 14, 2021

View reviewed changes

jiant/proj/simple/runscript.py Outdated Show resolved Hide resolved

Slight syntax change for simple runscript phases fix

bd488a8

jeswan approved these changes Apr 14, 2021

View reviewed changes

jeswan merged commit f796e5a into nyu-mll:master Apr 14, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

improve robustness of the simple runscript #1307

improve robustness of the simple runscript #1307

sakogan commented Apr 13, 2021

codecov bot commented Apr 13, 2021 •

edited

Loading

jeswan left a comment

sakogan commented Apr 14, 2021

improve robustness of the simple runscript #1307

improve robustness of the simple runscript #1307

Conversation

sakogan commented Apr 13, 2021

codecov bot commented Apr 13, 2021 • edited Loading

Codecov Report

jeswan left a comment

Choose a reason for hiding this comment

sakogan commented Apr 14, 2021

codecov bot commented Apr 13, 2021 •

edited

Loading